A method and controller for controlling a plurality of lighting devices

ABSTRACT

A method of controlling a plurality of lighting devices is disclosed. The method comprises obtaining a 360 degree panoramic image, rendering the 360 degree panoramic image at least partially on an image rendering device, obtaining positions of the plurality of lighting devices relative to the image rendering device, mapping the 360 degree panoramic image onto the plurality of lighting devices based on the positions of the plurality of lighting devices, such that each lighting device is associated with a part of the 360 degree panoramic image, determining, for each lighting device, a light setting based on an image characteristic of the part of the 360 degree image, and controlling each lighting device according to the respective light setting.

FIELD OF THE INVENTION

The invention relates to a method of controlling a plurality of lightingdevices. The invention further relates to a computer program product forexecuting the method. The invention further relates to a controller forcontrolling a plurality of lighting devices, and to a system comprisingthe controller and the plurality of lighting devices.

BACKGROUND

The number of connected lighting systems in home environments isincreasing. These connected lighting systems comprise lighting deviceswhich can be controlled individually, for example based on controlsignals received from a smartphone, a home automation system or a mediadevice such as a television. These lighting devices may further becontrolled based on content that is being rendered on a display.

International patent application WO 2010/061334 A1, for example,discloses controlling ambient light in a display system around a displaydevice based on depth information of an image, for example bycontrolling ambient light behind the display device based on the colorand intensity of the object or objects being far away from the viewerand by controlling ambient light in front of the display device based onthe color and intensity of the object or objects being close to theviewer.

International patent application WO 2017/174412 A1 discloses controllinga plurality of luminaires of a lighting system by capturing with animaging device an image of the luminaires, processing the image todetermine a location of each of the luminaires relative to a predominantfield of view of the user, and controlling the luminaires of thelighting system to emit illumination, based on their deter-minedlocations relative to the predominant field of view of the user. Theimage may be a panoramic image. For example, the panoramic image be a360° panoramic image. This can be used to capture the position anddirectionality of natural and artificial illumination sources, in orderto give a more accurate and perceptually similar representation of ascene image, such as an image of a “sunset over the ocean”. A user maythen select that scene image to be rendered by the luminaire and atleast two colors may be selected from the image. Respective locations ofthe at least two selected colors within that scene image are determined.Subsequently, locations of the selected colors within the scene imageare compared with the relative locations of the luminaires and/or theirillumination footprints within the physical space, such that for each ofthe locations in the scene image, a matching one of the luminaires isidentified. The colors of the scene image are mapped to the luminairesbased on the position of the luminaires relative to the user.

SUMMARY OF THE INVENTION

The inventors have realized that in existing systems, such as the systemof WO 2010/061334 Al1 and WO 2017/174412 A1, wherein lighting devicesare controlled based on image content displayed on a display device, thelighting control is limited to what is visible on the display device.If, for example, the display device displays a video of a sunset at thebeach, the lighting devices surrounding the viewer will be controlledbased on this information. A lighting device behind the display devicemay be controlled according to a light setting resembling the settingsun, and a lighting device in front of the display device may becontrolled according to the color of the sand in the image. Theinventors have realized that, in order to further immerse the viewer inthe image content, it would be desirable if the lighting devices do notonly represent image characteristics that are being displayed, but alsoadditional image characteristics, such as, referring to the previousexample, a boulevard parallel to the beach.

It is therefore an object of the present invention to create a moreimmersive user experience for a video display system.

According to a first aspect of the present invention, the object isachieved by a method of controlling a plurality of lighting devices, themethod comprising:

obtaining a 360 degree panoramic image,

rendering the 360 degree panoramic image at least partially on an imagerendering device,

obtaining positions of the plurality of lighting devices relative to theimage rendering device,

mapping the 360 degree panoramic image onto the plurality of lightingdevices based on the positions of the plurality of lighting devices,such that each lighting device is associated with a part of the 360degree panoramic image,

determining, for each lighting device, a light setting based on an imagecharacteristic of the part of the 360 degree image, and

controlling each lighting device according to the respective lightsetting.

By controlling a plurality of lighting devices based on imagecharacteristics (such as colors) of a 360 degree panoramic image,content that is not directly visible on the image rendering device (e.g.a display) is perceivable via the lighting devices. Thus, when a user iswatching a 360 degree video of a sunset at the beach, the sunset will bedisplayed on the display. Additional video content comprised in the 360degree video, such as a boulevard parallel to the beach, may not berendered on the display, but lighting devices surrounding the user maybe controlled based on this content. This is beneficial, because itcreates a more immersive user experience.

Another benefit of the present invention is that it enables a user tosee/perceive what is occurring in the rest of the 360 degree panoramicimage/video. If, for example, the user is watching a movie, and a partof a certain scene is being displayed on the display, for example twopeople having a conversation, fireworks may be exploding in a differentpart of the 360 degree panoramic video, for example behind the user.Even though the user cannot see the fireworks on the display, a lightingdevice behind the user may be controlled based on a part of the 360degree panoramic video that comprises the fireworks. This enables a userto experience the fireworks, even though the firework explosions are notrendered on the display.

The image rendering device may be a portable device, and the method mayfurther comprise: receiving an input indicative of a change oforientation of the image rendering device and maintaining the mapping ofthe 360 degree panoramic image onto the plurality of lighting devices. Auser may, for example, rotate/reorient the portable device, whereupon adifferent part of the 360 degree panoramic image may be rendered on theimage rendering device. The method may further comprise rendering a partof the 360 degree panoramic image as a function of the orientation ofthe portable device. This enables a user to view different parts of the360 degree image content by changing the orientation of the portabledevice. By maintaining the mapping of the 360 degree panoramic imageonto the plurality of lighting devices, the mapping does not change whena user reorients the portable device. For example, referring to theexample wherein a user watches a movie scene wherein two people arehaving a conversation and fireworks are exploding behind the user, theuser may rotate the portable device in order to see the fireworks on thedisplay of the portable device. Since the mapping of the 360 degreepanoramic image onto the plurality of lighting devices does not change,the lighting devices now in front of the user who is watching thefireworks on the portable device will still be controlled according tothe firework explosions.

The method may further comprise: receiving an input indicative of arotation of the 360 degree panoramic image relative to the imagerendering device and adjusting the mapping of the 360 degree panoramicimage onto the plurality of lighting devices in dependence on therotation. The input may, for example, be generated by a remote controldevice, by a device that controls the image rendering device, etc. Theinput may be a user input, or the input may be generated by a computerprogram running on a computing device. It may be desirable to rotate themapping, for example when a user wishes to see a different part of the360 degree panoramic image on the image rendering device. Additionallyor alternatively, the rotation may result in a better mapping ofdifferent image characteristics of the 360 degree panoramic image ontothe lighting devices.

The method may further comprise: analyzing the 360 degree panoramicimage to identify one or more dominant image characteristics andidentifying, in the 360 degree panoramic image, positions of the one ormore dominant image characteristics, and the mapping of the 360 degreepanoramic image onto the plurality of lighting devices and the imagerendering device may be further based on the positions of the one ormore dominant image characteristics, wherein the positions of the one ormore dominant image characteristics substantially correspond to thepositions of one or more of the plurality of lighting devices (and/orthe position of the image rendering device). By mapping the 360 degreepanoramic image onto the plurality of lighting devices and the imagerendering device based on one or more dominant image characteristics inthe 360 degree panoramic image, the mapping is being optimized such thateach dominant image characteristic is mapped onto a respective lightingdevice (and/or onto the image rendering device). This is beneficial,because each dominant image characteristic will be visible via therespective lighting device (and/or the image rendering device).

The method may further comprise analyzing the part of the 360 degreepanoramic image to identify a dominant image characteristic in the partof the 360 degree panoramic image, and determining the light setting fora respective lighting device based on the dominant image characteristic.By determining a dominant image characteristic (e.g. a dominant color, adominant object, etc.) in the part of the 360 degree panoramic image andby determining the light setting for the lighting device associated withthat part of the 360 degree panoramic image, the lighting device iscontrolled based on the dominant image characteristic.

The method may further comprise receiving a user position of a user or auser device, and the mapping of the 360 degree panoramic image onto theplurality of lighting devices may be further based on the user position.It may be beneficial to map the 360 degree panoramic image based on theposition of a user or a user device (operated by a user) in order tooptimize the user experience of the 360 degree panoramic image/videowatching experience. A user may, for example, be located in a first partof a space, and it may be esired that (only) lighting devices located inthat first space are controlled based on the 360 degree panoramic image.

The method may further comprise receiving a user orientation of a useror a user device, and the mapping of the 360 degree panoramic image ontothe plurality of lighting devices may be further based on the userorientation. In order to determine an initial mapping of the 360 degreepanoramic image onto the plurality of lighting devices, an (initial)orientation of the user or the user device may be determined, and themapping of the 360 degree panoramic image onto the plurality of lightingdevices may be based thereon.

The method may further comprise receiving capability information of eachof the plurality of lighting devices, which capability information isindicative of at least one of:

a color rendering capability of the respective lighting device, and

a number and/or a distribution of light sources of the lighting device,and the mapping of the 360 degree panoramic image onto the plurality oflighting devices may be further based on the capability information. Thecapability information may, for example, relate to a type of lightingdevice and its number and/or a distribution of light sources. A firstlighting device may, for example, be an LED strip, and a second devicemay, for example, be a spot lighting device. The 360 degree panoramicimage, for example an image of a sky with a sun, may be mapped onto theplurality of lighting devices such that the sun is mapped onto the spotlighting device, and the sky is mapped onto the LED strip. This furtheroptimizes the mapping of the 360 degree panoramic image, and therewiththe experience of watching the 360 degree panoramic image/video.

The 360 degree panoramic image may comprise depth information, and themapping of the 360 degree panoramic image onto the plurality of lightingdevices may be further based on the depth information. The 360 degreepanoramic image may, for example, comprise a depth map comprisinginformation relating to the distance of objects in the image relative tothe 360 degree panoramic image. A certain object present in the 360degree panoramic image may be mapped onto a lighting device that islocated at a distance from the 360 degree panoramic image thatcorresponds to the distance of the object relative to the 360 degreepanoramic image. This further optimizes the mapping of the 360 degreepanoramic image, and therewith the experience of watching the 360 degreepanoramic image/video.

The method may further comprise obtaining one or more positions of oneor more further image rendering devices, and the mapping of the 360degree panoramic image onto the plurality of lighting devices is furtherbased on the one or more positions of one or more further imagerendering devices. Additionally, a second part of the 360 degreepanoramic image, which is located at a position in the 360 degreepanoramic image that corresponds to a position of a further imagerendering device, may be rendered on the further image rendering device.Thus, additional to mapping the 360 degree panoramic image onto theplurality of lighting devices, the 360 degree panoramic image may bemapped onto further image rendering devices.

The 360 degree panoramic image may be a 360 degree panoramic video, andthe light setting may be based on a video characteristic of the part ofthe 360 degree image. The 360 degree panoramic video is a sequence of360 degree panoramic images. The plurality of lighting devices may becontrolled according to the sequence of 360 degree panoramic images overtime, such that the light output of the plurality of lighting devicescorresponds to the 360 degree panoramic video.

According to a second aspect of the present invention, the object isachieved by a computer program product for a computing device, thecomputer program product comprising computer program code to perform themethod of any one of the above-mentioned methods when the computerprogram product is run on a processing unit of the computing device.

According to a third aspect of the present invention, the object isachieved by a controller for controlling a plurality of lightingdevices, the controller comprising:

a communication unit configured to communicate with the plurality oflighting devices, and

a processor configured to obtain a 360 degree panoramic image being atleast partially rendered on an image rendering device, obtain positionsof the plurality of lighting devices relative to the image renderingdevice, map the 360 degree panoramic image onto the plurality oflighting devices based on the positions of the plurality of lightingdevices, such that each lighting device is associated with a part of the360 degree panoramic image, determine, for each lighting device, a lightsetting based on an image characteristic of the part of the 360 degreeimage, and to control, via the communication unit, each lighting deviceaccording to the respective light setting.

According to a fourth aspect of the present invention, the object isachieved by a lighting system comprising the controller, and a pluralityof lighting devices, each comprising a receiver configured to receivelighting control commands from the controller.

It should be understood that the claimed computer program product,controller and lighting system may have similar and/or identicalembodiments and advantages as the claimed method.

In the context of the present invention, the term “image characteristic”or “video characteristic” relates to an identifiable characteristic ofan image or a video. The characteristic may for example be a color, adominant color, an average color, an object such as a device,infrastructure, a plant, an animal or a person, an event such as a fire,fireworks an explosion, a sunrise, a natural phenomenon such as arainbow or the northern lights, etc.

BRIEF DESCRIPTION OF THE DRAWINGS

The above, as well as additional objects, image characteristics andadvantages of the disclosed systems, devices and methods will be betterunderstood through the following illustrative and non-limiting detaileddescription of embodiments of devices and methods, with reference to theappended drawings, in which:

FIG. 1 shows schematically an embodiment of a system for controlling aplurality of lighting devices;

FIG. 2 shows schematically examples of a 360 degree panoramic image;

FIG. 3a shows an example of a system for controlling a plurality oflighting devices and a display device based on a mapping of the 360degree panoramic image of FIG. 2;

FIG. 3b shows a top view of the system of FIG. 3 a;

FIG. 4a shows an example of a system for controlling a plurality oflighting devices and a display of a portable device based on a mappingof the 360 degree panoramic image of FIG. 2;

FIG. 4b shows a top view of the system of FIG. 4 a;

FIG. 5 shows an example of a system for controlling a plurality oflighting devices and a plurality of display devices based on a mappingof the 360 degree panoramic image of FIG. 2;

FIG. 6a shows an example of depth information of the 360 degreepanoramic image of FIG. 2;

FIG. 6b shows an example of a mapping of the 360 degree panoramic imageof FIG. 6a onto a plurality of lighting devices based on the depthinformation; and

FIG. 7 shows schematically a method of controlling a plurality oflighting devices.

All the figures are schematic, not necessarily to scale, and generallyonly show parts which are necessary in order to elucidate the invention,wherein other parts may be omitted or merely suggested. DETAILEDDESCRIPTION OF EMBODIMENTS

FIG. 1 shows schematically an embodiment of a system 100 for controllinga plurality of lighting devices 110, 112, 114. The system 100 comprisesan image rendering device 120 (e.g. a display, a projector, etc.) and acontroller 102, comprising a communication unit 106 for controlling theplurality of lighting devices 110, 112, 114. The controller 102 maycontrol the plurality of lighting devices 110, 112, 114 by communicatinglighting control commands to the plurality of lighting devices 110, 112,114. The lighting control commands may comprise control instruction tocontrol each of the plurality of lighting devices 110, 112, 114according to a light setting (e.g. light with a certain color,brightness and saturation). The controller 102 may be further configuredto communicate with the image rendering device 120. The communicationunit 106 may be configured to communicate with the lighting devices 110,112, 114 (and, optionally, the image rendering device 120) directly orindirectly, via any suitable wired or wireless communication protocol,for example via Ethernet, DMX, DALI, USB, Bluetooth, Wi-Fi, Li-Fi, 3G,4G or ZigBee.

The controller 102 further comprises a processor 104 (e.g. a microchip,circuitry, a microcontroller, etc.) configured to obtain a 360 degreepanoramic image being at least partially rendered on the image renderingdevice 120, to obtain positions of the plurality of lighting devices110, 112, 114 relative to the image rendering device 120, map the 360degree panoramic image onto the plurality of lighting devices 110, 112,114 based on the positions of the plurality of lighting devices 110,112, 114, such that each lighting device is associated with a part ofthe 360 degree panoramic image. The processor 104 is further configuredto determine, for each lighting device, a light setting based on animage characteristic of the part of the 360 degree image, and tocontrol, via the communication unit 106, each lighting device accordingto the respective light setting.

The processor 104 is configured to obtain a 360 degree panoramic image.The 360 degree panoramic image may be received from different sources,for example from a media player coupled to the image rendering device120, from a computer program such as a game, from video streamingsoftware, etc. The 360 degree panoramic image is at least partiallyrendered on the image rendering device 120. The controller 102 may beconfigured to render a part of the 360 degree panoramic image on theimage rendering device 120. Alternatively, a further device connected tothe image rendering device 120 may be configured to render the at leastpart of the 360 degree panoramic image on the image rendering device 120and to communicate the 360 degree panoramic image (or a stream of 360degree panoramic images) to the controller 102.

The processor 104 is further configured to obtain two-dimensional orthree-dimensional positions of the plurality of lighting devices 110,112, 114 relative to the image rendering device 120. The processor 104may be configured to receive position information of the plurality oflighting device 110, 112, 114, for example from the lighting devices,from an indoor positioning system, from a central control system, suchas a home automation system, etc. The position information may comprise,for example, a set of coordinates of a respective lighting device (e.g.x, y, z coordinates) relative to a space, or the position informationmay be descriptive of a position of a respective lighting device (e.g.front left, front right, rear left, rear right, center, etc.).Similarly, the processor 104 may be configured to receive positioninformation of the image rendering device 120 relative to the space.

The processor 104 is further configured to map the 360 degree panoramicimage onto the plurality of lighting devices 110, 112, 114 based on thepositions of the plurality of lighting devices 110, 112, 114, such thateach lighting device is associated with a part of the 360 degreepanoramic image. This is further explained in FIGS. 2-6, which aredescribed below. The processor may determine a (virtual) position andsize of the 360 degree panoramic image based on the positions of theplurality of lighting devices 110, 112, 114. The processor 104 may beconfigured to scale the 360 degree panoramic image based on thepositions of the plurality of lighting devices 110, 112, 114. Forexample, the processor 104 may increase the range of the 360 degreepanoramic image when the lighting devices are positioned further apartfrom the image rendering device 120. Additionally, the processor 104 maybe configured to map the 360 degree panoramic image onto the physicalspace. The processor 104 may, for example, receive information about thedimensions of the physical space and the positions of the lightingdevices therein. After the mapping, the processor 104 may associate eachof the lighting devices with a part of the 360 degree panoramic image.The processor 104 may determine an initial mapping of the 360 degreepanoramic image based on the positions of the plurality of lightingdevices 110, 112, 114 and/or based on a position of the image renderingdevice 120. The processor 104 may be further configured to adjust theinitial mapping based on one or more parameters, such as the position ofa user, the orientation of a user, the position of a dominant feature inthe 360 degree panoramic image, etc. Examples of adjustments of theinitial mapping are further described below.

FIG. 2 shows schematically examples of a 360 degree panoramic image. The360 degree panoramic image may, for example, be a cylindrical panoramicimage 200′ or a spherical panoramic image 200″. The top image 200 inFIG. 2 shows an unfolded version of the cylindrical panoramic image 200′or the spherical image 200″. The 360 degree panoramic images in FIG. 2show an image of a vehicle 202, a sun 204, a tree 206, grass 208 and asky 210.

The processor 104 is further configured to determine, for each lightingdevice, a light setting based on an image characteristic of the part ofthe 360 degree panoramic image. The image characteristic (or in someembodiments video characteristic) relates to an identifiablecharacteristic of the 360 degree panoramic image (or a video). Thecharacteristic may for example be a color, a dominant color, an averagecolor, an object such as a device, infrastructure, a plant, an animal ora person, an event such as a fire, an explosion, a sunrise, etc.

The parts of the 360 degree panoramic image that are associated with thelighting devices may be predefined, or the processor 104 may beconfigured to determine the parts, for example by using known imageanalysis techniques to identify the characteristics. The processor may,for example, divide the 360 degree panoramic image in multiple parts andassociate at least some of the parts with lighting devices. The 360degree panoramic image may be divided into equal parts, or be dividedinto parts based on characteristics of the 360 degree panoramic image.Referring to FIG. 2, the image may, for example, be divided in 5 partsbased on the characteristics of the image 200: a first part comprisingthe car 202, a second part comprising the sun 204, a third partcomprising the tree 206, a fourth part comprising the grass 208 and afifth part comprising the sky 210. After the creation of the parts ofthe 360 degree panoramic image, the processor 104 may associate at leastsome of the parts with lighting devices. The first part comprising thecar 202 may for example be associated with a first lighting device, thesecond part comprising the sun 204 with a second lighting device, thethird part comprising the tree 206 with a third lighting device, thefourth part comprising the grass 208 with one or more fourth lightingdevices located on floor level and the fifth part comprising the sky 210with one or more fifth lighting devices located on ceiling level.

The processor 104 may be configured to extract light settings for eachlighting device based on the part of the 360 degree panoramic image thatis associated with that lighting device. The processor 104 may, forexample, extract a color from the part of the 360 degree panoramic imageby determining a dominant or average color in the part of the 360 degreepanoramic image, and generate a light setting based on that color.Additionally or alternatively, the processor 104 may recognize an objectin the part of the 360 degree panoramic image by using known imageanalysis techniques, and generate a light setting based on that a colorof the object.

The processor 104 may be further configured to analyze the 360 degreepanoramic image to identify one or more dominant image characteristics,and identify, in the 360 degree panoramic image, positions of the one ormore dominant image characteristics, and map the 360 degree panoramicimage onto the plurality of lighting devices and the image renderingdevice based on the positions of the one or more dominant imagecharacteristics, wherein the positions of the one or more dominant imagecharacteristics substantially correspond to the positions of one or moreof the plurality of lighting devices (and/or the position of the imagerendering device). The processor 104 may, for example, calculate aplurality of different mappings of the 360 degree panoramic image ontothe plurality of lighting devices 112, 114, 114 (and the image renderingdevice 102), determine for each mapping a number of matches betweenlighting devices (and the image rendering device) and dominant imagecharacteristics, and select a mapping form the plurality of differentmappings based on the number of matches of the mappings, for example byselecting a mapping which has a number of matches above a thresholdvalue, or by selecting a mapping which has the highest number ofmatches.

FIGS. 3a and 3b show an example of a system 300 for controlling aplurality of lighting devices 310, 312, 314, 316 and a display 320 basedon a mapping of the image 200. The processor (not shown) of thecontroller (not shown) may obtain the positions of the lighting devices310, 312, 314, 316 relative to the display 320 and map the image 200onto the lighting devices 310, 312, 314, 316 based on the positions ofthe lighting devices 310, 312, 314, 316 (and, optionally, based on theposition of the display 320). The processor may further associate a partof the image 200 with each of the lighting devices 310, 312, 314, 316(and, in embodiments wherein the processor is further configured to mapthe image 200 onto the display 320, associate a part of the image 200with the display 320). In the example of FIG. 3a , the vehicle 202 maybe rendered on the display 320. Based on the mapping of the image 200onto the lighting devices 310, 312, 314, 316, a first part of the image200 comprising the sun 204 may be associated with lighting device 310,and a second part of the image 200 comprising the tree 206 may beassociated with lighting device 316. A third and fourth part of theimage 200 (for example the grass 208 in between the vehicle 202 and thesun 204, and the grass 208 in between the vehicle 202 and the tree 206,respectively) may be associated with lighting devices 312 and 314,respectively. Based on the association between the parts of the imagesand the lighting devices, the processor may determine light settings forthese lighting devices. The processor may determine a first lightsetting (for example a bright yellow light setting resembling the sun204) for lighting device 310 based on the first part of the image 200,determine a second light setting (for example a dark green light settingresembling the tree 206) for lighting device 316 based on the secondpart of the image 200 and determine a third and fourth light setting(for example a light green light setting resembling the grass 208) forlighting devices 312 and 314 based on the third and fourth parts of theimage 200. The processor may then control the lighting devices 310, 312,314, 316 according to these light settings.

In the example of FIGS. 3a and 3b , the processor has mapped the 360degree panoramic image onto the lighting devices such that the positionsof the image rendering device and the lighting devices determineposition and the outer perimeter of the 360 degree panoramic image. Inother words, the 360 degree panoramic image is mapped such that theimage rendering device is positioned at an outer perimeter of the 360degree panoramic image. Alternatively, as illustrated in the system 400of FIGS. 4a and 4b , the processor may map the 360 degree panoramicimage onto the lighting devices such that the positions of the imagerendering devices create the outer perimeter of the 360 degree panoramicimage. The position of the image rendering device may, for example, beused to determine the center point/point of origin of the 360 degreepanoramic image relative to the plurality of lighting devices.

The image rendering device 120 may be comprised in a portable device(e.g. a smartphone, a tablet pc, a wearable device such as smartglassesor a smartwatch, etc.). The controller 102 may be further comprised inthe portable device. The processor 104 may be further configured toreceive an input indicative of a change of orientation of the imagerendering device 120/portable device, and maintain the (initial) mappingof the 360 degree panoramic image onto the plurality of lighting devices110, 112, 114. A user may, for example, rotate/reorient the portabledevice, whereupon a different part of the 360 degree panoramic image maybe rendered on the image rendering device 120. The processor 104 mayrender a part of the 360 degree panoramic image as a function of theorientation of the portable device. This enables a user to viewdifferent parts of the 360 degree image content by changing theorientation of the portable device. By maintaining the mapping of the360 degree panoramic image onto the plurality of lighting devices 110,112, 114, the mapping does not change when a user reorients the portabledevice.

FIGS. 4a and 4b show an example of a system 400 for controlling aplurality of lighting devices 410, 412, 414, 416 and a display 420 basedon a mapping of the image 200. The processor (not shown) of thecontroller (not shown) may obtain the positions of the lighting devices410, 412, 414, 416 relative to the display 420 and map the image 200onto the lighting devices 410, 412, 414, 416 based on the positions ofthe lighting devices 410, 412, 414, 416 (and, optionally, based on theposition and the orientation of the display 420). In the example ofFIGS. 4a and 4b , the display 420 is comprised in a portable devicecarried by a user 402. The processor may further associate a part of theimage 200 with each of the lighting devices 410, 412, 414, 416 (and, inembodiments wherein the processor is further configured to map the image200 onto the display 420, associate a part of the image 200 with thedisplay 420). In the example of FIG. 4a , the vehicle 202 may berendered on the display 420. Based on the mapping of the image 200 ontothe lighting devices 410, 412, 414, 416, a first part of the image 200comprising the sun 204 may be associated with lighting device 410, and asecond part of the image 200 comprising the tree 206 may be associatedwith lighting device 416. A third and fourth part of the image 200 (forexample the grass 208 in between the vehicle 202 and the sun 204, andthe grass 208 in between the vehicle 202 and the tree 206, respectively)may be associated with lighting devices 412 and 414, respectively. Basedon the association between the parts of the images and the lightingdevices, the processor may determine light settings for these lightingdevices. The processor may determine a first light setting (for examplea bright yellow light setting resembling the sun 204) for lightingdevice 410 based on the first part of the image 200, determine a secondlight setting (for example a dark green light setting resembling thetree 206) for lighting device 416 based on the second part of the image200 and determine a third and fourth light setting (for example a lightgreen light setting resembling the grass 208) for lighting devices 412and 414 based on the third and fourth parts of the image 200. Theprocessor may then control the lighting devices 410, 412, 414, 416according to these light settings.

The user 402 may reorient 404 the portable device (and therewith thedisplay 420). The processor may maintain the mapping of the image 200onto the plurality of lighting devices 410, 412, 414, 416. The processormay further a part of the 360 degree panoramic image as a function ofthe orientation of the portable device. The user may, for example,rotate 404 the portable device (and therewith the display 420) towardslighting device 410, whereupon the processor may render the sun 204 ofthe image 200 onto the display 420. Since the mapping of the image ontothe plurality of lighting devices 410, 412, 414, 416 has not changed,the lighting devices 410, 412, 414, 416 are still controlled based ontheir initial mapping. Thus, lighting device 416 will still emit darkgreen light resembling the tree 206, even though the tree 206 is alsorendered on the display 420.

The processor 104 may be further configured to receive an inputindicative of a rotation of the 360 degree panoramic image relative tothe image rendering device 120. The processor 104 may be furtherconfigured to adjust the mapping of the 360 degree panoramic image ontothe plurality of lighting devices 110, 112, 114 in dependence on therotation. The input may, for example, be generated by a remote controldevice, a rotary knob, or a remote control object comprising anorientation sensor, or a smartphone. Additionally or alternatively, theinput may be generated by a device that controls the image renderingdevice 120. The input may be a user input, or the input may be generatedby a computer program running on a computing device, for example by gamesoftware, wherein the rotation is a gameplay element of the game. It maybe desirable to rotate the mapping, for example because a user wishes tosee a different part of the 360 degree panoramic image on the imagerendering device 120. Beneficially, the rotation may result in a bettermapping of different image characteristics of the 360 degree panoramicimage onto the lighting devices 110, 112, 114. Referring to FIG. 3, auser may provide an input to rotate the image 200 relative to thedisplay 320. As a result, the processor may adjust the mapping of theimage onto the plurality of lighting devices 310, 312, 314, 316. Theuser may, for example, rotate the image 200 counterclockwise. As aresult, the tree 206 may be rendered on the display 320, and lightingdevice 310 may be controlled according to a color of the vehicle 202,lighting device 316 may be controlled according to the bright yellowcolor associated with the sun 204 and lighting devices 312 and 314 may(still) be controlled according to the light green light settingresembling the grass 208.

The processor 104 may be further configured to analyze the 360 degreepanoramic image to identify one or more dominant image characteristics,and, upon receiving the input indicative of the rotation of the 360degree panoramic image, map the 360 degree panoramic image onto theimage rendering device 120 and the plurality of lighting devices 110,112, 114 such that one of the one or more dominant image characteristicsis rendered on the image rendering device 120. As a result, the 360degree panoramic image ‘snaps’ to the image rendering device 120.Additionally or alternatively, the processor 104 may be configured tomap the 360 degree panoramic image onto the image rendering device 120and the plurality of lighting devices 110, 112, 114 such that one of theone or more dominant image characteristics is rendered on a lightingdevice. As a result, the 360 degree panoramic image ‘snaps’ to thatlighting device when it is being rotated relative to the plurality oflighting devices 110, 112, 114. This ensures that dominant imagecharacteristics are continuously visible, and will not be ‘lost’ inbetween the lighting devices 110, 112, 114 and the image renderingdevice(s) 120. In other words, during the rotation of the 360 degreepanoramic image, a dominant image characteristics may only move to anext lighting device/image rendering device if a proper match between adominant image characteristic and a next lighting device/image renderingdevice has been found. This means that dominant image characteristicsremain ‘magnetically snapped’ to matching lighting devices/imagerendering device(s) until the rotation change has reached a pointwhereby in the mapping the dominant image characteristic has come closeenough to transition to a next lighting device/image rendering device.The transition from a first light setting to a second light setting mayoccur instantly or gradually (e.g. a smooth/fading transition of thefirst light setting to the second light setting).

The processor 104 may be further configured to receive a user positionof a user or a user device, and map the 360 degree panoramic image ontothe plurality of lighting devices based on the user position. Theprocessor 104 may determine an initial mapping of the 360 degreepanoramic image based on an initial position of the user. The positionof the user or the user device (such as a smartphone, smartglasses,etc.) may be obtained from an (indoor) positioning system or from theuser device. The processor 104 may be configured to map the 360 degreepanoramic image (only) onto lighting devices located within a(predefined) proximity of the image rendering device 120 (or within a(predefined) proximity of the user or the user device). A user may, forexample, be located in a first area of a space comprising the lightingsystem 100. The processor 104 may therefore map the 360 degree panoramicimage onto lighting devices that are located in the first area. If theuser would move to a second area of the space, the processor 104 may mapthe 360 degree panoramic image onto lighting devices located in thesecond area. In another example, a user may be positioned close to theimage rendering device 120 (e.g. a television). The processor 104 may beconfigured to map the 360 degree panoramic image (only) onto lightingdevices located within a (predefined) proximity of the image renderingdevice 120 (or within a (predefined) proximity of the user).

The processor 104 may be further configured to receive a userorientation of a user or a user device, and map the 360 degree panoramicimage onto the plurality of lighting devices based on the userorientation. The processor 104 may determine an initial mapping of the360 degree panoramic image based on an initial orientation of the user.The orientation of the user or the user device (such as a smartphone,smartglasses, etc.) may be obtained from an (indoor) positioning systemor from the user device. The processor 104 may for example be configuredto map the 360 degree panoramic image (only) onto lighting deviceslocated within the field of view of the user. Additionally oralternatively, when the user orientation changes, the processor 104 mayrotate the mapping of the 360 degree panoramic image accordingly.

The processor 104 may be further configured to receive capabilityinformation of each of the plurality of lighting devices 110, 112, 114via the communication unit 106. The capability information of a lightingdevice may be indicative of a color rendering capability of thatlighting device, a number and/or a distribution of light sources of thelighting device, etc. The processor 104 may be further configured to mapthe 360 degree panoramic image onto the plurality of lighting devices110, 112, 114 based on the capability information. The processor 104may, for example, analyze the 360 degree panoramic image to identify(dominant) image characteristics, calculate a plurality of differentmappings of the 360 degree panoramic image onto the plurality oflighting devices 112, 114, 114 (and the image rendering device 102) bymapping the (dominant) image characteristics onto the lighting devicesbased on their capability information, determine for each mapping anumber of matches between lighting devices (and the image renderingdevice 120) and image characteristics of the 360 degree panoramic imageand select a mapping form the plurality of different mappings based onthe number of matches of the mappings, for example by selecting amapping which has a number of matches above a threshold value, or byselecting a mapping which has the highest number of matches. Theprocessor 104 may, for example, receive capability information about twolighting devices of the lighting system 100. A first lighting device 110may be a spotlight configured to emit white light positioned on ceilinglevel and a second lighting device 112 may be an LED strip configured toemit colored light positioned on ground level. The 360 degree panoramicimage, for example an image of a sky with a sun, and the processor 104may map it onto the two lighting devices such that the sun is mappedonto the spotlight, and the sky is mapped onto the LED strip.

The system 100 may further comprise one or more further image renderingdevices. The processor 104 may be further configured to obtain one ormore positions of one or more further image rendering devices. Thepositions of the one or more further image rendering devices may forexample be received from an (indoor) positioning system or from thefurther image rendering devices. The processor 104 may be furtherconfigured to map the 360 degree panoramic image onto the plurality oflighting devices further based on the one or more positions of one ormore further image rendering devices. The processor 104 may, forexample, analyze the 360 degree panoramic image to identify (dominant)image characteristics, calculate a plurality of different mappings ofthe 360 degree panoramic image onto the image rendering devices,determine for each mapping a number of matches between image renderingdevices and image characteristics of the 360 degree panoramic image andselect a mapping form the plurality of different mappings based on thenumber of matches of the mappings, for example by selecting a mappingwhich has a number of matches above a threshold value, or by selecting amapping which has the highest number of matches. FIG. 5 illustrates anexample of a system 500 wherein the image 200 has been mapped onto aplurality of image rendering devices 520, 522 (displays) and lightingdevices 510, 512. A processor (not shown) may obtain positions of thedisplays 520, 522 and lighting devices 510, 512, analyze the image 200to identify image characteristics (the vehicle 202, the sun 204, thetree 206, the grass 208 and the sky 210), calculate different mappingsonto the displays 520, 522 and the lighting device 510 and select one ofthe mappings, for example a mapping wherein the vehicle 202 is mappedonto a first display 520, the tree 206 is mapped onto a second display522, the sun 204 is mapped onto a first lighting device 510 and the skyis mapped onto a second lighting device 512. Based on this mapping, thedisplays 520, 522 and lighting devices 510, 512 are controlledaccordingly.

The 360 degree panoramic image may comprise depth information, forexample a depth map. The processor 104 may be further configured to mapthe 360 degree panoramic image onto the plurality of lighting devices110, 112, 114 further based on the depth information. The 360 degreepanoramic image may, for example, comprise a depth map comprisinginformation relating to the distance of objects in the image relative tothe 360 degree panoramic image. The processor 104 may, for example,determine a base distance from a reference point (e.g. a user, a userdevice, the display device, etc.) and determine a distance of an objectin the 360 degree panoramic image relative to the base distance based onthe depth information. The processor 104 may further compare thepositions of the plurality of lighting devices 110, 112, 114 with thepositions of the objects in the 360 degree panoramic image in order todetermine how to map the 360 degree panoramic image onto the lightingdevices, such that the positions of the lighting devices (substantially)correspond to the positions of the objects in the 360 degree panoramic(depth) image. The processor 104 may calculate different mappings, andselect one of the mappings wherein the positions of the lighting devices(substantially) correspond to the positions of the objects in the 360degree panoramic (depth) image.

FIG. 6a illustrates an example of two-dimensional depth information 600of the image 200. In this example, the vehicle 202 may have a distance602 from the base 600, the sun 204 may have a distance 604 from the base600 and the tree 206 may have a distance 606 from the base 600. FIG. 6billustrates a mapping of the image 200 onto a plurality of lightingdevices 612, 614, 616. The processor (not shown) may calculate aplurality of mappings, and determine one mapping wherein positions ofthe lighting devices 612, 614, 616 substantially correspond to thepositions of the objects 202, 204, 206 in the image 200.

The 360 degree panoramic image may be a 360 degree panoramic video, andthe light setting may be based on a video characteristic of the part ofthe 360 degree image. The 360 degree panoramic video is a sequence of360 degree panoramic images. The processor 104 may control the pluralityof lighting devices according to the sequence of 360 degree panoramicimages over time, such that the light output of the plurality oflighting devices corresponds to the 360 degree panoramic video,according to any one of the above-mentioned ways/methods.

FIG. 7 shows schematically a method 700 of controlling a plurality oflighting devices 110, 112, 114. The method 700 comprises: obtaining 702a 360 degree panoramic image, rendering 704 the 360 degree panoramicimage at least partially on an image rendering device 120, obtaining 706positions of the plurality of lighting devices relative to the imagerendering device 120, mapping 708 the 360 degree panoramic image ontothe plurality of lighting devices 110, 112, 114 based on the positionsof the plurality of lighting devices 110, 112, 114, such that eachlighting device is associated with a part of the 360 degree panoramicimage, determining 710, for each lighting device, a light setting basedon an image characteristic of the part of the 360 degree image andcontrolling 712 each lighting device according to the respective lightsetting.

The method 700 may be executed by computer program code of a computerprogram product when the computer program product is run on a processingunit of a computing device, such as the processor 104 of the controller102.

It should be noted that the above-mentioned embodiments illustraterather than limit the invention, and that those skilled in the art willbe able to design many alternative embodiments without departing fromthe scope of the appended claims.

In the claims, any reference signs placed between parentheses shall notbe construed as limiting the claim. Use of the verb “comprise” and itsconjugations does not exclude the presence of elements or steps otherthan those stated in a claim. The article “a” or “an” preceding anelement does not exclude the presence of a plurality of such elements.The invention may be implemented by means of hardware comprising severaldistinct elements, and by means of a suitably programmed computer orprocessing unit. In the device claim enumerating several means, severalof these means may be embodied by one and the same item of hardware. Themere fact that certain measures are recited in mutually differentdependent claims does not indicate that a combination of these measurescannot be used to advantage.

Aspects of the invention may be implemented in a computer programproduct, which may be a collection of computer program instructionsstored on a computer readable storage device which may be executed by acomputer. The instructions of the present invention may be in anyinterpretable or executable code mechanism, including but not limited toscripts, interpretable programs, dynamic link libraries (DLLs) or Javaclasses. The instructions can be provided as complete executableprograms, partial executable programs, as modifications to existingprograms (e.g. updates) or extensions for existing programs (e.g.plugins). Moreover, parts of the processing of the present invention maybe distributed over multiple computers or processors.

Storage media suitable for storing computer program instructions includeall forms of nonvolatile memory, including but not limited to EPROM,EEPROM and flash memory devices, magnetic disks such as the internal andexternal hard disk drives, removable disks and CD-ROM disks. Thecomputer program product may be distributed on such a storage medium, ormay be offered for download through HTTP, FTP, email or through a serverconnected to a network such as the Internet. A listing of the entire setof pending claims is submitted herewith per 37 CFR 1.121. This listingof claims will replace all prior versions, and listings, of claims inthe application.

1. A method of controlling a plurality of lighting devices, the methodcomprising: obtaining a 360 degree panoramic image, rendering the 360degree panoramic image partially on an image rendering device, obtainingpositions of the plurality of lighting devices relative to the imagerendering device, mapping the 360 degree panoramic image onto theplurality of lighting devices based on the positions of the plurality oflighting devices relative to the image rendering device, such that eachlighting device is associated with a part of the 360 degree panoramicimage, determining, for each lighting device, a light setting based onan image characteristic of the part of the 360 degree image, andcontrolling each lighting device according to the respective lightsetting, wherein the method further comprises receiving an inputindicative of a rotation of the 360 degree panoramic image relative tothe image rendering device, and adjusting the mapping of the 360 degreepanoramic image onto the plurality of lighting devices in dependence onthe rotation.
 2. The method of claim 1, wherein the image renderingdevice is a portable device, and wherein the method further comprises:receiving an input indicative of a change of orientation of the imagerendering device, and maintaining the mapping of the 360 degreepanoramic image onto the plurality of lighting devices.
 3. (canceled) 4.The method of claim 1, wherein the input is a user input received via auser interface.
 5. The method of claim 1, further comprising: analyzingthe 360 degree panoramic image to identify one or more dominant imagecharacteristics, and identifying, in the 360 degree panoramic image,positions of the one or more dominant image characteristics, wherein themapping of the 360 degree panoramic image onto the plurality of lightingdevices and the image rendering device is further based on the positionsof the one or more dominant image characteristics, wherein the positionsof the one or more dominant image characteristics substantiallycorrespond to the positions of one or more of the plurality of lightingdevice.
 6. The method of claim 1, further comprising: analyzing the partof the 360 degree panoramic image to identify a dominant imagecharacteristic in the part of the 360 degree panoramic image, anddetermining the light setting for a respective lighting device based onthe dominant image characteristic.
 7. The method of claim 1, furthercomprising receiving a user position of a user or a user device, whereinthe mapping of the 360 degree panoramic image onto the plurality oflighting devices is further based on the user position.
 8. The method ofclaim 1, further comprising receiving a user orientation of a user or auser device, wherein the mapping of the 360 degree panoramic image ontothe plurality of lighting devices is further based on the userorientation.
 9. The method of claim 1, further comprising receivingcapability information of each of the plurality of lighting devices,which capability information is indicative of at least one of: a colorrendering capability of the respective lighting device, and a numberand/or a distribution of light sources of the lighting device, whereinthe mapping of the 360 degree panoramic image onto the plurality oflighting devices is further based on the capability information.
 10. Themethod of claim 1, wherein the 360 degree panoramic image comprisesdepth information, and wherein the mapping of the 360 degree panoramicimage onto the plurality of lighting devices is further based on thedepth information.
 11. The method of clalm 1, further comprising:obtaining one or more positions of one or more further image renderingdevices, wherein the mapping of the 360 degree panoramic image onto theplurality of lighting devices is further based on the one or morepositions of one or more further image rendering devices.
 12. The methodof claim 1, wherein the 360 degree panoramic image is a 360 degreepanoramic video, and wherein the light setting is based on a videocharacteristic of the part of the 360 degree image.
 13. A computerprogram product for a computing device, the computer program productcomprising computer program code to perform the method of claim 1 whenthe computer program product is run on a processing unit of thecomputing device.
 14. A controller for controlling a plurality oflighting devices, the controller comprising: a communication unitconfigured to communicate with the plurality of lighting devices, and aprocessor configured to obtain a 360 degree panoramic image beingpartially rendered on an image rendering device, obtain positions of theplurality of lighting devices relative to the image rendering device,map the 360 degree panoramic image onto the plurality of lightingdevices based on the positions of the plurality of lighting devicesrelative to the image rendering device, such that each lighting deviceis associated with a part of the 360 degree panoramic image, determine,for each lighting device, a light setting based on an imagecharacteristic of the part of the 360 degree image, and to control, viathe communication unit, each lighting device according to the respectivelight setting, wherein the processor is further configured to receive aninput indicative of a rotation of the 360 degree panoramic imagerelative to the image rendering device, and to adjust the mapping of the360 degree panoramic, image onto the plurality of lighting devices independence on the rotation.
 15. A lighting system comprising: thecontroller of claim 14, and a plurality of lighting devices, eachcomprising a receiver configured to receive lighting control commandsfrom the controller.